Guess I don't see any harm here, but isn't it pretty much recommended to always aggregate at the statsd layer prior to graphite/graphana/whatever first?

We actually had, for usegalaxy.org, statsd->graphite at first, but are now using statsd->influxdb->grafana for a similar display.

I would very much like to see some admin docs here how to set this up and why. It might be trivial for us but we should be building this for others.

We actually had, for usegalaxy.org, statsd->graphite at first, but are now using statsd->influxdb->grafana for a similar display.

We're using the graphite-influxdb interface. It's a bit of work to reconfigure influxdb due to our deployment choices, so this was the easiest method for us.

@martenson not sure how much there is to document. What would you want to see? I would guess that most people who use this are people who already know what graphite/statsd/etc are and are using it elsewhere.

The galaxy side setup is literally just filling out these vars. Setting up graphite/statsd/influxdb/etc are really probably outside the scope of possible docs.

@erasche I guess in this case your explanation makes sense. But having an (short) admin page about options and config values to change if you want to track performance of your instance is something I would applaud to.

@martenson #3968

Side note, would you be willing to share any grafana dashboard you have built? I, and the community, I'm sure would love to see what metrics you all find important to monitor.

@erasche We do not have anything too useful at our instance yet. It has been deployed when we had UI slowness on Main to track it down, but we figured it out before grafana started to have data. And then it was down for a while so it has no data now I think. :/

@martenson That's incorrect, it has (lots of) data. We just need to figure out what we need to track want to render.

@erasche it's an item on our roadmap (#1928) to figure out what timings we care about, build dashboards, etc. Happy to work with you on that and share whatever we come up with, of course.

@dannon That is great news! I was under the impression that we lost the data in the downtime.

I did not set up a graphite/carbon server to test this live, but code is short and looks solid. Thanks @erasche